The present invention relates generally to computer vision systems and digital image processing, and more particularly to the digital extraction of an image from its background.
To produce certain special visual effects, it is often desirable to separate a foreground image from a background image. One method for accomplishing this task is referred to as xe2x80x9cblue-screeningxe2x80x9d and involves placing a blue or other fixed-color screen (e.g. a green screen) behind the desired foreground object, typically the image of a person (the colors blue and green hue that strongly contrasts with most colors of human skin). In blue-screening, the system checks to see which pixels in the input image are not blue and labels those pixels as foreground pixels. The system can then composite, or blend, the object (i.e. the collection of all foreground pixels) onto a destination background image.
One of the disadvantages of using blue-screening for object compositing is that it requires a fixed color screen behind the object. Another disadvantage is that if any of the colors on the object, such as an item of clothing, are blue, xe2x80x9cholesxe2x80x9d will appear in the object in the destination image. This occurs because the pixels in the blue areas on the object will not be labeled as foreground pixels and thus will not be composited with the rest of the object, resulting in the object having holes when composited onto the destination image. Another disadvantage is, obviously, the need for a special room or screen to provide the appropriate background color.
Other prior art background subtraction procedures, from the field of computer vision, are used to eliminate the fixed color screen requirement. One procedure involves building an average background image by taking a predetermined number of sample images of a typically multi-colored background and creating a background model. For each new sample image taken, each pixel in the new image is compared to its corresponding pixel in the background model being formed. Changes are assumed to be to the foreground object, i.e. this is done to determine whether the pixel in the current sample image is a foreground (xe2x80x9cobjectxe2x80x9d). Pixels that are determined to be part of the foreground are then blended or composited onto the destination image.
One disadvantage with this procedure is if a foreground pixel happens to match its corresponding background model pixel color, it will not be considered a foreground pixel. This will introduce holes into the composited object. Another disadvantage is that shadows cast by the object often make the object, when composited, appear to have its original form plus extraneous appendages (as a result of the shadows). This is because the procedure may mistakenly label the xe2x80x9cshadowxe2x80x9d pixels as foreground pixels.
Yet another disadvantage is that if any portion of the background changes or if the camera is moved while the background model is being built, certain portions of the background (e.g. the portions that moved) will be incorrectly labeled as part of the foreground and be composited onto the destination image. Although there are prior art techniques for updating the background model to reflect changes, they cannot account for a constantly changing background such as one that includes a changing television screen or a window looking out onto a busy street.
As is apparent from the foregoing discussion, it has been difficult to obtain a clean-edged outline for a foreground image being extracted from a background image. One known method of creating a clean-edged outline is hand painting the outline for the background subtraction which is typically done by an application developer. While this is appropriate in the application design stage, it is not appropriate for an end user to have to hand paint or manually create the outline of the image.
The present invention creates a clean-edged extraction of a foreground image from its background in an automated and effective manner. More particularly, the present invention includes an improved method of background subtraction wherein a clean image of the object being composited or texture mapped is created through improvements in determining the outline of the object. As a result, the present invention reduces noise near the edges of an object in an input image from which the background is being xe2x80x9csubtracted.xe2x80x9d The resulting image can be used for example, in a texture map application or for compositing.
In the texture mapping process any errors that may occur in the background subtraction or inaccuracies that arise from the background subtraction will typically remain throughout the duration of the process. The present invention provides a closed curve which is essentially a noise-reduced outline of the object that is being mapped or composited. By virtue of this process there are no holes or gaps within the object image when it is mapped or composited, since all pixels within the closed curve are reconsidered part of the object.
A background subtraction apparatus of the present invention includes a key point locator for locating key points on a known object type, a boundary point locator for locating boundary points of the known object that make up the edges of the known object, and an edge processor for processing the edges to provide a clean-edged extraction of the known object from a background image. Preferably, the key point locator includes an alignment detector for detecting alignment of an image of the known object type with a skeleton image. Still more preferably, the skeleton image is an exoskeleton image and the known object type is a human being.
A method for subtracting a background in accordance with the present invention includes locating key points on a known object type, locating boundary points of the known object that comprise edges of the known object, and processing the edges to provide a clean-edged extraction of the known object from a background image. Preferably, locating the key points include detecting the alignment of an image of the known object type with a skeleton image. Even more preferably, the key points include primary key points at extremities of an image of the human being and secondary key points near a torso of an image of the human being.
These and other advantages of the present invention will become apparent upon a study of the specification and drawings of this application.